Over the last decade, neural networks have attracted an ever increasing interest as signal processors. Such processors, essential characteristics reside in the non-linear mapping of input signals onto output signals using spatially distributed elements and massively parallel information processing in a densely interconnected structure. Such a processor is a robust and fault-tolerant processing device. Furthermore, neural networks can be programmed by means of training on the basis of examples rather than by means of prescribing algorithmic instructions such as in the field of conventional microprocessors. Training can be accomplished with or without a supervisor, the latter version being called "self-learning". Neural networks are expected to play an important role, especially in the fields of associative memories, classifiers, pattern recognition and optimization problems. For a general introduction, see: "An Introduction to Computing with Neural Nets", R. P. Lippmann, IEEE ASSP Magazine, April 1987, pp. 4-22.
Although some of the mathematical algorithms created for neural network control or neural network simulation appear to be useful in seeking software solutions to particular problems, the real benefit of the neural network approach is believed to be obtainable only by way of dedicated hardware. A variety of electronic and optical hardware implementations of neural networks have seen the light over the last decade. For example, the Laboratoires d'Electronique de Philips has developed the first commercially electronic, fully digitally operating, trainable neural chip for modular neural network architectures, see U.S. Pat. No. 4,994,982. An electronic implementation of an analog neural net is known from, for instance, U.S. Pat. No. 4,866,645. Optical neural nets are dealt with in: "Optical Neural Computers", Y. S. Abu-Mostafa and D. Psaltis, Scientific American 256, March 1987, pp. 66-73.
Often, spatially tight designs are required to realize a sufficiently large number of the neurons and synapses in a confined space. A trainable neural net requires additional circuitry to individually and accurately modify the weights in the net's learning phase, necessarily leading to an increase of the system's size and complexity. Since the processing capability of a neural net increases with larger numbers of neurons and interconnections, nowadays efforts in neural net technology find a major target in increasing the density of neurons and interconnections. As an example, a major problem encountered in the design of electronic embodiments is the limitation to the number of neurons or neuron functionalities that can be implemented on a single chip. Typically, the maximum number of neurons or neuron functionalities that can be integrated into a chip using state of the art technology is well lower than one thousand. Furthermore, the complexity and the amount of additional components needed to implement the learning capability of a net containing such large numbers of neurons is expected to be at least proportionally higher.
The examples mentioned above have in common that the associated architectures primarily have a lumped character. That is, the various respective functional elements of the neural net which perform the basic computations, such as the neurons and the synapses, are implemented by physically distinct devices or circuits. These devices and circuits are dedicated to operations such as weight multiplication, summation and applying a sigmoid function.
One of the ways to reduce of the number of neurons or neuron layers, to simplify the interconnection scheme and to increase efficiency, is to employ high-order terms (squares, products, cubes, etc.) of the input signals supplied to the neural net. These high-order terms then serve as the signals to be handled by the neurons. See, for instance, C. Lee Giles et at., "Learning, Invariance and Generalization in High-Order Neural Nets", Appl. Opt., Vol. 26, 1987, pp. 4972-4978. The high-order dependencies are determined in advance and can be thought of as pre-processed input data, the pre-processing being based on some prior knowledge of the kind of problems that the neural net is meant to solve. For example, different classification problems to be handled by a neural net may require different forms of non-linear correlations between the input signals.